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ABSTRACT 



This paper suggests a variety of topics for the National 
Assessment of Vocational Education (NAVE) to support as it reviews progress 
under the Carl D. Perkins Vocational Technical Education Act 1998 (Perkins 
III) . Section 1 covers the task most closely related to the legislation 
itself, which is to examine how community colleges and other postsecondary 
institutions have responded to the key requirements of Perkins III. The 
requirements are: development of accountability measures; incorporation of 
academic competencies into post secondary occupational education (PSOE) and 
the integration of academic and occupational education; tech prep efforts to 
link high schools and community colleges; allocation of federal funds; and 
types of institutions providing PSOE. Section 2 addresses five criteria to 
judge the effectiveness of PSOE programs, each of which serves as a starting 
point for at least two kinds of research: determining what conventional or 
modal practices in PSOE are and identifying promising practices for others to 
emulate. The criteria are labor market information and connections; 
curriculum and pedagogy; support services; connections to other programs; and 
data and improvement. Section 3 examines four issues beyond Perkins III: 
meaning of completion; "new fluidity" in postsecondary education; limited 
success of federal legislation; and the role the federal government should 
perform. (YLB) 
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In many ways postsecondary occupational education (PSOE) is a stepchild — even a stepchild of a stepchild. 
The institutions where it takes place-community colleges, technical institutes, some area vocational schools, 
other public training centers, private proprietary schools — are not well known and are often of low status. In 
federal policy, these institutions are often afterthoughts: they do not benefit from the large programs (Chapter 
1, aid for bilingual and handicapped students) aimed at K-12 schooling; they are not the principal targets of the 
large amounts of federal funds in postsecondary student aid; and federal aid for vocational education, one of 
the few federal programs providing funding to both secondary and postsecondary programs, has always been 
written with secondary education in mind. And within the institution where most PSOE occurs — community 
colleges — occupational education often suffers again from second-class status relative to academic or transfer 
education. 

But PSOE is more important than its status suggests. It is now the locus of most vocational education, as least 

vocational education understood as preparation for immediate employment below the baccalaureate level, ^ 
since vocational education in high schools has either been transformed into more general programs ("education 
through occupations" or "college and careers") or has dissipated into random course offerings. The majority of 
students in community colleges — about 60 percent^ — are in occupational programs; almost all the rest are 
enrolled for broadly occupational purposes, even if they think these will be realized through academic 
programs and transfer. PSOE fulfills the promise of providing postsecondary education — increasing viewed as 
necessary for both individual mobility and employer needs — for the large numbers of occupations for which 
baccalaureate degrees are not necessary, and indeed the economic returns to subbaccalaureate education are 
highest for those who complete occupational programs and find related employment (Grubb 1999). The 
promises of "lifelong learning," so often empty rhetorical flourishes, are probably best realized within PSOE, 
with its large number of older students returning for skill upgrading or retraining. And when comprehensive 
institutions like community colleges provide PSOE, their "bridging function"-their ability to create bridges or 
paths from one purpose to another — involves PSOE in a web of other programs and purposes. In this role, 
students who enter community colleges for remedial purposes, "experimenters" looking for a life path, welfare 
recipients, dislocated workers, or others in job training and adult education programs can find a way into the 

mainstream of economic life.^ 

If PSOE is more important than is often recognized, then it merits more research than is currently available. 
Like PSOE itself, research on this comer of American education has been badly neglected. Most of the research 
on vocational education concentrates on the secondary level; most of the research on community colleges 
focuses on transfer or academic education, or on the high-status "economic development" role of providing 
training for particular firms. And so PSOE has rarely had its day in the sun.^ I therefore begin (rather than end) 
this paper with the researcher's conventional plea: more research is needed. 




BEST COPY AVAILABLE 



2 



In this paper I suggest a variety of topics for the National Assessment of Vocational Education to support as it 
reviews progress under the Carl Perkins Vocational and Applied Technology Education Amendments of 1998, 
or Perkins m. The task most closely related to the legislation itself is to examine how community colleges and 
other postsecondary institutions have responded to the key requirements of Perkins m, and I cover these in 
Section I. They include the development of accountability measures; the incorporation of academic 
competencies into PSOE and the integration of academic and occupational education; tech-prep efforts to link 
high schools and community colleges; the allocation of federal funds; and the types of institutions providing 
PSOE. 

However, while federal legislation for occupational education has addressed several important ways of 
improving PSOE, it has never been especially comprehensive, particularly in its approach to postsecondary 
issues. In Section II I therefore shift to a variety of topics intended to address the effectiveness of PSOE, using 
a five-part conception of the institutional conditions necessary for effectiveness as well as proposing certain 
research related to outcomes. This section is consistent with the broader purpose underlying federal legislation 
for vocational education, which has always stressed program improvement. 

In Section III I examine four issues that are perhaps more important than any of the others presented in this 
paper: the question of who counts as "completers" in PSOE, almost certainly the most vexing question to 
PSOE administrators; the "new fluidity" in postsecondary education and its potential threat to established 
programs; the question of why federal legislation in many areas related to vocational education and training has 
often had unsystematic effects; and the correlative question of what role the federal government should 
perform. These issues are not raised directly in Perkins HI, and these issues are not included in the 
congressional mandate to NAVE. But they represent the most pressing current developments in PSOE and the 
central dilemmas in federal policy toward PSOE. Therefore ignoring them may consign NAVE to irrelevance 
and the eighth circle of hell, 6 and can only ensure that federal legislation remains out of touch with current 
problems. 

In Section IV I draw together the various issues presented in the first three sections into research strategies. 
Typically, each kind of research encompasses several issues; for example, the case studies of community 
colleges I propose could encompass a variety of issues, including local practices about integration, 
accountability, and tech prep as well as participation with other education and training providers. Furthermore, 
the various research strategies are related to one another: a broad sample of providers could respond to 
questionnaire surveys; a smaller sample of these institutions could become case studies; and a smaller sample 
of these could be the starting point for "community case studies" examining the variety of programs in a 
community. I will therefore cross-reference the research presented in Section IV to the issues presented in 
Sections I-HI, and vice versa. 

Finally, since I propose too many issues for NAVE to support, I briefly present my own priorities for research 
in Section V. 

Too often, proposals for research become little more than laundry lists of topics with little coherence, and 
certainly this paper could be critiqued for precisely that flaw. However, there is one central issue underlying 
virtually all of the research proposed in this paper: how do we make postsecondary occupational education 
more effective? This question is addressed in a few ways in Perkins III (Section I), though the approaches to 
effectiveness could be broadened, as I argue in Section II. And failure to consider some basic issues of current 
conditions will mean that future legislation is not especially powerful in improving effectiveness, which is why 
the topics in Section in are so important. 

The stress on effectiveness is also, in my view, the way to think about equity in PSOE. The current equity 
problems are not, with a few exceptions, those of access to PSOE, since community colleges and other 
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institutions providing PSOE are usually open-access institutions with low tuition, low opportunity costs, and 
high levels of both state and federal subsidies. The problems experienced by the groups who are the usual focus 
of equity efforts-low-income students, minorities, women seeking access to nontraditional employment, older 
students, dislocated workers, handicapped students, and other "nontraditional" students are instead those of 
movement through these programs, of completion rather than noncompletion, of the coherence of learning and 
its value in the labor market. While there are some issues specific to equity and particular groups, particularly 
those considering support services (in Section n.3), the greatest improvement in equity would come simply 
from improving the quality of PSOE for all students. 



n 

Finally, the issue of effectiveness is one way of creating a compelling vision, or policy narrative/- for 
postsecondary occupational education. In secondary vocational education, such a compelling vision has 
emerged in the "new vocationalism" (or "education through occupations," as I prefer to label it, or David 
Stem's conception of "college and careers") — in which academic and occupational competencies are 
integrated, in which school-based and work-based learning are linked, in which high schools are thoroughly 
restructured into smaller units (academies, clusters, or pathways), and in which broader forms of occupationally 
oriented education lead to further education as well as employment. But a comparable vision has not yet 
emerged for postsecondary occupational education, aside from the vision underlying most federal 
postsecondary policy — which has been simply to expand access through student aid and, more recently, tax 
credits. A compelling complement (not alternative) to the vision of access is a vision of effectiveness — and 
this idea underlies much of the research proposed in this paper. 

I. The Issues in Current Federal Legislation 

The current legislation contains several specific provisions that are particularly germane to PSOE. They can be 
investigated in various ways, including questionnaires to postsecondary institutions (the basis for the previous 
NAVE). However, because questionnaire results often seem unreliable, I recommend that they be 
supplemented in virtually every issue with case studies, to understand better how the intentions of Perkins III 
are being met (or ignored). 

LI: Accountability and Performance Measures 

Many observers feel that the most novel element of Perkins III is its requirement that states develop at least 
four measures of state performance. States that do not make adequate progress in their own performance 
measures are threatened with having their state plans disapproved and their funding eliminated (Section 121e) 
and, in addition, incentive funding under the Workforce Investment Act (WLA) is allocated on the basis of 
performance measures for Perkins m, job training, and adult education. The final consequence of the state 
performance measures adopted is that local programs are to be evaluated by the "State adjusted levels of 
performance," and those not making substantial progress are required to develop local improvement plans 
(Section 123). Unlike the former legislation, therefore, there might be real consequences to poor performance 
under these new performance measures. I note that the development of these accountability systems requires 
both state measures, used for comparison across states, and local measures, used to identify low-performing 
local programs and improve their performance so state measures can improve. While local indicators should be 
tied in some way to state measures, they could be different, and the sanctions a state uses with local program 
could go beyond the improvement plan required in Perkins IE. 

The new provision generates a variety of obvious questions. The first is what state measures the different states 
have adopted, and then what local indicators they have adopted — questions that can be resolved by examining 
state plans (see IV. 1). However, if performance measures are to be used to improve the quality of local 
programs, a more interesting issue is what states have done first to require or encourage local programs to meet 
local standards, including what forms of technical assistance or additional financing they have provided in 
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addition to threats like potential reductions in funding. This question can be addressed both from the vantage of 
the state, through questionnaires or interviews with state directors (see IV. 2), and from the vantage of the local 
providers, through questionnaire studies (IV.3). Finally, the most important issue is how these performance 
measures and standards have influenced local programs, where it's important to recognize that these influences 
might be unintended and negative-like the "creaming" of students and the cooking of data sometimes seen in 
JTPA programs — as well as intended and positive. While questionnaires to local programs may elicit some of 
these changes (see IV.3), corroboration from local case studies (IV.4) would be necessary, particularly in 
uncovering some of the less positive potential effects of accountability. 

I note that most of the four core measures required by Perkins III involve issues raised in other research 
proposed in this paper. The first core measure, related to academic and technical competencies, could be 
reflected by passage of licenses or industry-generated standards, for example (see IH.l). The second measure, 
on completing postsecondary degrees or credentials, involves the complex issue of who "completers" are (see 
m.l), and some states may have used measures of completion other than certificates or associate's degrees. 
(Many colleges have begun to define early-exit options, partly for welfare recipients, that recognize earlier 
stages of "completion.") The third core measure, related to placement and retention in employment, could most 
readily be addressed with the kinds of Unemployment Insurance (UI) data reviewed briefly in Section H.6. 
Therefore, information about accountability measures being adopted may shed some light on other PSOE 
issues. 

Some local programs, particularly technical institutes and community colleges, are also likely to be subject to 
performance measures required by WIA, either for adult services or for adult education. A question related 
more to the relationship among federal programs (see II.5) is how Perkins performance measures interact with 
WIA measures, or whether there are inconsistencies that make compliance with various indicators difficult. 
Again, this issue can be investigated to some extent through local questionnaires (IV.3), though case studies of 
local programs (IV.4) and community case studies examining the interaction among programs (IV.5) would be 
valuable in corroborating the responses of local institutions. 

One question raised with the new performance requirements is whether the costs of complying with Perkins III 
outweigh the benefits to PSOE-particularly in those states that have made little progress since Perkins II in 
developing data systems that can be used for accountability. (This is a new version of an old complaint about 
federal policy: "the tail wags the dog," in the sense that small amounts of federal funding have disproportionate 
influences on local programs.) Federal funding accounts for a small proportion of PSOE and of most local 
programs, especially in comprehensive institutions like community colleges (see 1.4). Some institutions may 
decide that the burdens of participating in performance measures outweigh the benefits of small sums; 
therefore research should be alert for indications that local programs have forgone federal resources. This issue 
is in turn related to the question of what role the federal government should play in PSOE (III.4): if the federal 
government hands out trivial sums of money and requires inordinate costs for compliance, then its help on 
balance may be no help at all. 

1.2: Incorporating Academic Competencies and Integrating Academic and Occupational 
Education 

Perkins HI is filled with language exhorting and requiring PSOE to incorporate a range of academic, 
occupational, and technical competencies, and the first of eight requirements of local uses of funds specifically 
provides for the "integration of academics with vocational and technical education through a coherent sequence 
of courses." It must be acknowledged that the purpose of stimulating broader and more integrated programs, 
through this and other provisions in Section 135 (including the "all aspects of an industry" provision), are 
accompanied by language permitting local programs to spend federal funds on a wide variety of activities, so 
that local programs can comply with federal requirements without paying the least amount of attention to this 



requirement for improvement (see III.3). Nonetheless, one of the central changes of Perkins n, continued in 
Perkins m, was the attempt to move away from the kind of "narrow vocationalism" often critiqued by the 
business community (e.g., Committee for Economic Development 1985, p. 15) toward a broader and more 
integrated conception of PSOE, parallel to a similar effort at the secondary level. 

The most obvious task is to learn more about what local institutions have done to implement this provision. 

The prior NAVE conducted surveys of local programs to ascertain changing practices, and such questions 
could easily be included in questionnaires to local institutions (see IV.3). In my view, the earlier study 
generated local responses that were exaggerated; for example, 38.5 percent of institutions reported starting 
tandem courses — coordinated academic and occupational courses — before 1991-92, even though all my 
other experience with community college indicates that such courses are quite rare. s (See also Badway 1998, 
for a more extended argument and evidence that the prior NAVE had a low threshold for compliance with the 
integration requirements in Perkins II.) One possible interpretation is that, since the previous surveys were done 
quite soon after Perkins II was enacted, they represented hopes and dreams rather than reforms already 
implemented. Now, however, with an additional six years for implementation, it should be easier to see what 
local programs have put in place. 

However, the possibility for exaggeration is still high, particularly because these surveys are completed by 
central administrators whose knowledge of what happens in classrooms is usually somewhere between limited 
and zero (Grubb and Associates 1999, Ch. 8). Therefore, any replication of this survey should be 
complemented by case studies of local colleges and other providers (see IV.4) to see precisely how academic 
competencies are incorporated or integrated into PSOE. These local case studies should ideally include 
interviews with occupational deans and instructors to get their perspectives on such integration, on what has 
been accomplished, and on the (inevitable) barriers to integration. However, while self-assessments of teaching 
practices are reasonably accurate under some circumstances, they are invariably exaggerations (Mayer 1999) 
and they fail to convey precisely what happens within classrooms-as the observation-based writing of Grubb 
and Associates (1999, Ch. 7) and Perrin (1998) illustrates. Therefore, the case studies should include 
observations of occupational classes and integrated classes to examine the extent and form that integration of 
academic competencies take. 

Some community colleges collect good longitudinal evidence on their students, and potentially this could be 
used to examine the effects of integration on progress and subsequent success (see IV. 1 1). For example, where 
some occupational classes are linked with academic classes and some are not, it may be possible to compare 
the subsequent success of students in the two different settings. While obvious problems of selection and 
self-selection may be present, NAVE researchers ought at least be alert to opportunities for further analyses of 
local data. 

1.3: Tech Prep 

Starting in Perkins n, tech prep is a vision of linking the last two years of high school with two years of 
community college, where the links could (and should) include the reform and integration of curricula at the 
two levels as well as mechanisms to enhance the transition between the two. At least in theory, tech prep is a 
mechanism that would encourage more "noncollege bound" students to attend postsecondary education, and to 
enroll without the need for remedial education that has become so pervasive. 

However, like many of the aspects of work force development programs, the term "tech prep" has taken on 
several meanings among local educators: sometimes it refers to efforts to integrate academic and vocational 
education, sometimes to simple articulation agreements between high schools and colleges, and sometimes to 
school-to-work programs incorporating some work-based learning. There's been a substantial amount of 
research already on tech-prep programs, including a national evaluation (Hershey Silverberg, Haimson, Hudis, 
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and Jackson 1998) as well as many studies by NCRVE (e.g., Bragg, Layton, and Hammons 1994; Bragg 1996; 
Grubb, Badway, Bell, and Kraskouskas 1996); Debra Bragg is now in the midst of an evaluation of the effects 
of tech prep, supported by NCRVE. Most of these find incomplete implementation of tech prep, with local 
programs usually incorporating components (like articulation agreements or integrated curricula at the high 
school level) but very few developing truly integrated programs. 

Another finding, particularly important for postsecondary institutions, is that the most substantial changes have 
been at the high school level, with changes (including curricular changes) in community colleges relatively less 
common. In the early days of tech prep this was due to the fact that students ostensibly in tech-prep programs 
had not yet reached community colleges; even in Bragg's current research, the number of completers is still 
small. Three research strategies seem obvious. One is to continue Bragg's research, now scheduled to end this 
year, as a way of gathering more information about the eventual completion (or noncompletion) of students in 
the postsecondary component of tech-prep programs. Another is to include examination of tech-prep programs 
in the institutional case studies (IV.4) to learn more about the curriculum developments (if any) and the barriers 
at the postsecondary level; the marginal cost of such research would be small. The final strategy is to 
incorporate some questions into the questionnaire surveys of community colleges (IV.3), parallel to those asked 
in the previous NAVE; as is true for the integration of academic and occupational education, the earlier results 
constitute a benchmark and it will now be possible to see how much progress there has been and what kinds of 
postsecondary institutions seem to have made the most progress. 

However, it’s difficult (at least for me) to imagine that further research on tech prep will lead to powerful new 
insights. The finding of incomplete implementation of tech prep has emerged over and over again, and is likely 
to be rediscovered yet one more time. Therefore, I suspect that the most interesting question to be asked of tech 
prep programs is why federal legislation has so often led to partial reforms and "business as usual," even when 
the underlying ideas are widely celebrated (see IV.3). 

1.4: Where Does the Money Go? 

A basic question to be addressed in NAVE is the question of how federal funds under Perkins III are spent. 

This can be addressed in several ways: by examining state plans and supporting fiscal documents (IV. 1), and by 
examining both the pattern of spending and the process by which decisions about Perkins funds are made in 
local institutions, through case studies (IV.4). 

In the past, a great deal of Perkins funds have been spent on normal expenditures and routine reforms-new 
equipment, for example, or updating courses in ways that might be thought to be the responsibility of local 
programs using state and local resources. This is explicitly allowed in Perkins III, which permits funds to be 
spent to "initiate, improve, expand, and modernize quality vocational and technical education programs," but it 
is contrary to the intent of program improvement in any more fundamental sense. One potential reason why this 
has been true is that most local public institutions are funded through formulas that seem to have academic 
programs in mind. There is no recognition for the variation in costs among programs, and no routine allocation 
for capital expenditures; thus high-cost occupational programs, and those with extensive requirements for 
up-to-date equipment and workshops, must often grovel for extra money and must use federal funds for what 
seem like routine expenditures of keeping programs up to date. One way to examine whether this common 
complaint explains the uses of federal funds is to examine whether the patterns of spending, and the process by 
which federal funds are allocated, are different in states with cost differentials in funding and explicit funding 
for capital equipment, or whether the patterns and processes are different in institutions like technical institutes 
with funding dedicated to occupational programs, compared to patterns in comprehensive community colleges. 
This is also part of examining why federal funds for occupational education and other work force development 
programs have so often been ineffective in fostering improvement (see IV.3). 
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Another dimension of the flow of federal funds is how large they are relative to other funds spent in local 
institutions. In both questionnaires (IV.3) and case studies (IV.4), the relative importance of Perkins funds can 
be ascertained. Then one question is whether institutions with relatively high fractions of their funds from 
Perkins use the revenues in ways different from those with relatively low levels of funding. 

This research could take the opportunity to track all federal funds into a variety of postsecondary institutions. 
This can be done both through case studies (IV.4) and questionnaires (IV.3) completed by fiscal officers, at 
least to determine the variety of federal funds from student aid as well as other sources (like the Fund for the 
Improvement of Postsecondary Education or the National Science Foundation). Grants and loans should also be 
examined from the federal perspective, using federal data (e.g., from the National Postsecondary Student Aid 
Surveys) on the amounts of grants and loans to community colleges and technical institutes (see IV.8). One 
puzzle that has never been fully answered is why community colleges and technical institutes receive such 
relatively small fractions of grants and loans, compared with the numbers of students (or low-income students) 
they serve (e.g., Grubb and Tuma 1991), and a specific investigation of this issue might generate 
recommendations of how to change the administration of grants and loans-at either the federal or local levels-to 
facilitate the flow of these funds to PSOE students. 

Finally, an enormous new source of federal funds was "appropriated" in 1996 with the Hope Tax Credits and 
the Lifelong Learning Tax Credits, amounting to perhaps $6 billion per year. The amounts provided through 
the Hope Tax Credits were pegged to average community college tuition levels, as if these credits could enable 

all students to attend at least two years of postsecondary education free of charge. 2 However, tax credits are 
awkward and potentially inequitable mechanisms of funding, particularly to low-income students whose taxes 
may not be sufficient to allow tax credits to be of any use. Therefore a separate study should try to estimate the 
flow of these two tax credits to students in sub-baccalaureate postsecondary institutions generally and in PSOE 
in particular. Alternative methodologies may be necessary: one is to survey students to see what proportion of 
those in community colleges and technical institutes (and in four-year colleges) applied for these tax credits; 
from the federal perspective, IRS records can provide information on the total amounts of these tax credits 
claimed and they can in theory identify the types of institutions attended since taxpayers qualifying for these 
credits must identify the institutions they attended. The distribution of these tax credits is a sufficiently broad 
issue that NAVE might be able to join forces with other research organizations-for example, the College Board, 
with its continuing interest in the funding of postsecondary education-to examine these funding patterns. 

Federal funding for PSOE is particularly complex because the funds are partly direct expenditures to 
institutions (e.g., Perkins HI and FIPSE), partly direct expenditures to students (grants and loans), and partly tax 
expenditures to students (in Hope and Lifelong Learning Tax Credits). One question that should be 
incorporated in any overall examination of the federal role in PSOE (see IV.4) is whether this mix of 
funding-inadvertent and unplanned-is the most effective and equitable way to support PSOE. 

1.5: Types of Postsecondary Institutions 

While the majority of PSOE, and of funds through Perkins HI, is surely spent in community colleges, there are 
many other postsecondary institutions that provide PSOE, many of these poorly understood — except, perhaps, 
by those within them. These include technical colleges and institutes, which provide two-year associate's 
degrees but are (unlike comprehensive community colleges) generally confined to occupational subjects;^ area 
vocational schools (the Regional Occupation Programs and Centers in California, the BOCES in New York), 
which often provide various certificates to adults; postsecondary programs operated by local educational 
authorities, which include adult schools in many states and some vocational schools and career centers; some 
vocational centers operated primarily with job training funds that may also receive some Perkins funds and 
provide certificates similar to those offered in community colleges or area vocational schools; and other 
hybrids and consortia too varied to name. Perkins IH recognizes institutions of higher education (community 
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colleges and technical institutes), LEAs providing postsecondary education, area vocational schools, 
postsecondary institutions operated by the Bureau of Indian Affairs to any Indian tribe, educational service 
agencies, and consortia (Sec. 3 (10)). 

A useful exercise would be to catalogue the variety of these postsecondary institutions, to determine the 
amounts of federal funds going to each, the enrollments and types of credentials granted, and the trends in these 
enrollments. The sources of data could include national data sets like NPSAS and the Beginning Postsecondary 
Surveys, state enrollment and expenditures data, and federal sources on the allocation of Perkins funds. There's 
a boundary problem to confront: the types of institutions considered will differ if the starting point is (a) the 
universe of institutions receiving Perkins funds; (b) the institutions reported in a particular national data set, 
like NPSAS or BPS; or (c) data collected from states on their definitions of postsecondary programs. One task, 
therefore, would be to examine how various ways of identifying these institutions affects the numbers and 
types of institutions considered. 

To examine trends, however, data over time would be necessary. It appears, for example, that technical 
institutes are converting to comprehensive community colleges,^ as part of a larger historical process of 
replacing specialized institutions with comprehensive ones. Area vocational schools also seem relatively 
unimportant, except perhaps in certain states that have maintained systems of them; in other cases they have 
been upgraded into technical institutes. But the information on which my observations are based are random 
and incomplete; a more systemic study could identify the patterns throughout the country more accurately. 

While the descriptive purposes of examining enrollments and funding patterns are useful, a purpose more 
consistent with the theme of effectiveness is to examine the types of programs offered. There are certainly 
some highly sophisticated programs being offered by these varied institutions, like the two-year programs in 
technical fields offered by the Texas State Technical Colleges. On the other hand, many area vocational schools 
and career centers appear to offer short (e.g, 15-week) programs for relatively unskilled occupations, quite 

similar to the short-term job training programs offered by JTPA that have proved so ineffective. 12 More 
generally, the economic benefits of short PSOE programs are varied and uncertain (Grubb 1999), at least 
compared to longer programs and associate's degrees. Any study of the varied types of postsecondary 
institutions should try to uncover data on their effectiveness-either by finding local follow-up studies or 
perhaps state UI wage record data (see H.6 below) — to see whether more can be learned about the labor 
market value of such programs. In the absence of more specific state or local data, however, my concern is that 
institutions providing short-term training are relatively ineffective. 

1.6: Substate Allocation of Perkins Funds 

One persistent aggravation among PSOE administrators is that federal legislation requires states to allocate 
Perkins funds according to the number of students receiving Pell grants (Section 132), presumably in an effort 
to allocate more funding to institutions with low-income students. However, the number of Pell recipients is 
not necessarily a good indicator either of the number of students in PSOE or of the concentration of 
low-income students, partly because Pell grant recipients include many academic students and therefore don't 
reflect needs for PSOE; partly because many PSOE students are part-time and therefore ineligible for Pell 
grants; partly because many of them are older and do not appear to apply for grants and loans; and partly 
because the rates of receiving Pell grants may be related to factors — e.g., the amount of information provided 
to students, the extent of assistance with student aid, the sophistication of students in dealing with institutions 
— that have nothing to do with the needs for PSOE. One study might therefore examine alternative methods of 
allocating funds, to see what difference it makes to the overall distribution and to weigh the consequences of 
different allocation formulas. Alternatives might include, for example, the number of students in PSOE, where 
enrollments in PSOE could be defined in various ways; the total enrollments in local institutions, on the 
assumption that Perkins aid should be potentially available to all students in subbaccalaureate institutions; the 
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number of PSOE students weighted by local income, to allocate more resources to institutions in low-income 
areas; the number of PSOE students weighted by the local unemployment rate, to distribute more funds to low 
employment areas; or some measure of the size of the sub-baccalaureate labor market, to tie funding to the 
extent of appropriate employment rather than enrollments. Each of these (and other) measures would change 
the distribution of funds and the incentives on local programs. This information would also be useful in 
considering the potential direction that federal policy toward PSOE might take (see IV .4). 



H. Beyond Perkins III: Five Principles for Effective PSOE Programs 



Since the early 1960s, one of the central purposes of federal aid for vocational education has been program 
improvement. Most recently, in Perkins II and now in Perkins III, improvement has principally been defined as 
the integration of academic and occupational education, the creation of tech-prep programs, and the 
development of performance measures to reflect outcomes. However, there are many more dimensions to 
effectiveness than these three particular approaches, and NAVE ought to consider research related to 
effectiveness and program improvement that goes beyond the conception embedded in Perkins III. 

In examining effectiveness, there is often no substitute for analyzing outcomes, and I propose several ways to 
research outcomes in Section H.6. However, often it is impossible to measure outcomes. In other cases, 
changes need to be considered before long-run outcome measures can be available, and in any event evidence 
of poor outcomes is literally useless without some sense of what kinds of improvements might enhance 
outcomes. Therefore it is almost always necessary to have some idea of what elements of PSOE programs are 
necessary for greater effectiveness, in addition to (and sometimes instead of) information about outcomes. Over 
the last several years I have developed five criteria to judge the effectiveness of many types of occupational 
education and training programs, based on my reading of the available evaluation and econometric literature 

and on the characteristics of particularly effective programs.- 1 ^ Others may differ about whether these are 
precisely the right criteria and about how important each of them is, but they at least provide a starting point for 
thinking about effectiveness — and the potential research that might illuminate improved practice. Each of 
these five criteria therefore serves as a starting point for at least two kinds of research: determining what 
conventional or modal practices in PSOE are, and identifying promising practices for others to emulate. 

II. 1: Labor Market Information and Connections 

Effective programs understand the local labor market, and target those jobs that are likely to employ 
individuals with community college credentials, with relatively high earnings, strong employment growth, and 
opportunities for individual advancement. Some community colleges do this through careful local needs 
assessments; others, particularly those with extensive cooperative programs, accomplish this by maintaining 
stable links with local employers. Programs in high employment areas, and many programs during the current 
economic boom, can get away with ignoring this recommendation, of course, though probably at the expense of 
long-run effectiveness. But those colleges that fail to consider the hiring requirements and the quality of jobs 
for which they provide occupational preparation are likely to place individuals in minimum-wage positions 
with few prospects for advancement, with dismal results over the long run. 

There are various mechanisms by which local institutions can keep abreast of labor market patterns and trends, 
including formal labor market assessments, advisory committees, the local contacts of individual instructors, 
placement office, follow-up studies, and — most impressive of all — the contacts afforded by high-quality 
cooperative education programs. However, the extent of these efforts varies substantially, and some-for 
example, advisory committees that meet once a year for ceremonial functions, or placement offices that stress 
low-quality "stay-in-school" jobs, or instructor contacts that are uneven and idiosyncratic-are not much use at 
all (Grubb 1996b, Ch. 6). Thus, contact with local labor market conditions is far from automatic in PSOE, and 
may need improvement in many cases. 
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Two kinds of research are possible. One is to ascertain how the mechanisms that potentially connect local 
programs with employers and labor markets actually function, through local case studies (see IV.4). An adjunct 
to such a study would try to identify best practices from around the country, to provide guidance to community 
colleges and other PSOE providers about the most effective ways of maintaining connections with local 
employers. 

The second and complementary study would question employers about their practices-particularly about their 
proclivity to hire from local PSOE programs rather than other sources, their preferences for experience over 
formal schooling, the kinds of promotion practices they use and the role of formal PSOE in promotion. Some 
observers have concluded that many employers — at least those at the sub-professional level-hire on the basis 
of skills attained regardless of the source of those skills, so that community colleges students compete with 
those trained in the military, those who have learned through experience, and others who have picked up skills 
in informal or self-taught ways. In other cases, licensing requirements, company policy, and formal or informal 
agreements with local colleges create tighter linkages between formal PSOE and employment. Further analysis 
of employer practices — through either survey methods, interviews (see IV.8), or the analysis of several local 
labor markets (see IV.5) — would help clarify where employment practices encourage rather than discourage 
hiring from local PSOE programs. 

11.2: Curriculum and Pedagogy 

Effective programs contain an appropriate mix of academic (or remedial, or basic) education, occupational 
skills, and work-based learning, in the best cases integrated with one another. In addition, while flexible and 
innovative scheduling may be necessary, the intensity of both academic and vocational education must be 
appropriate to the jobs being targeted. Finally, effective programs pay attention to the pedagogy of everything 
they teach, whether classroom-based or work-based. 

This principle, which really embodies three interrelated conditions for effectiveness, clarifies the obvious point 
that PSOE is an educational program. If the fundamental conditions for learning are not present, then PSOE 
can at best serve as a signal of motivation, ability, or some other valued characteristic but cannot make students 
more productive. Research under this principle is potentially vast, since there is virtually no analysis in this 
country of the effectiveness of different approaches to curriculum and instruction (Achtenhagen and Grubb 
1999). One strand includes the research on the incorporation of academic competencies proposed above in 
Section 1.2. 

Another strand could include more focused research on work-based learning, and its integration with other 
components of instruction. Despite the recent interest in this country in work-based learning, and the small 
amounts of funding from the School-to-Work Opportunities Act, work-based learning is comparatively rare in 
most community college (except in certain fields like nursing). The questionnaire-based surveys of community 
colleges provide one opportunity to ask about the extent of work-based learning, though such results would no 
doubt simply replicate those available from earlier studies (e.g., the Bragg studies cited earlier). The case 
studies (see IV.4) are probably more effective ways to ascertain the role of work-based learning, including the 
resources available to support it, the nature of its integration (if any) with the rest of the curriculum, and 
(potentially) measures of effectiveness if the case study institutions have adequate data on student outcomes. 

The other topic that has been virtually unexplored in the area of curriculum and pedagogy is the nature of 
teaching in PSOE. Aside from one chapter describing occupational classes in Richardson, Fisk, and Okun 
(1983) and an analysis of occupational teaching in several chapters of my recent book (Grubb and Associates 
1999), there’s been nothing written about the nature and challenges of teaching in postsecondary occupational 
subjects. (For a broader review of the international literature, see Achtenhagen and Grubb 1999.) From the 
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work that has been done, I have argued that occupational teaching is in many ways more complex than 
academic teaching: it incorporates a greater variety of competencies or "intelligences"; more settings, typically 
including classrooms, workshops, and sometimes job placements; different and often more demanding forms of 
reading, writing, and mathematics than are taught in conventional academic classes; a greater number of 
literacies including specialized diagrams and charts as well as conventional writing and mathematics; and 
distinctly different approaches to teaching that I have labeled "skills" versus "systems" approaches, paralleling 
the distinctions between behaviorist and constructivist teaching in other subjects. Yet for all this complexity, 
PSOE instructors receive virtually no preparation in teaching (save in Iowa and a few isolated community 
colleges), and most postsecondary instructors learn how to teach through trial and error. The classroom 
observations I have recommended as part of institutional case studies (see IV.4) could readily be extended into 
a special study of its own. This special study might, for example, concentrate on PSOE teaching within selected 
occupational areas, or within occupational areas subject to licensing or to industry-generated standards (see 
IV. 1). or within institutions where instructor training is practiced. In any of these forms, its central purpose 
would be to further investigate the nature of pedagogy in PSOE, the origin of different instructors' approaches 
to teaching, and the potential ways of improving instruction. 

Finally, I should acknowledge the difficulty of addressing questions of the appropriate intensity for PSOE 
programs. The dismal outcomes of job training programs are due in substantial part to their limited duration 
and intensity; within PSOE, the economic returns are clearly higher to those completing associate's degrees 
than to individuals with certificates taking shorter amounts of time or those with varying amounts of 
coursework. If more is better, then how much is enough? And when is more too much? One way to examine 
this question is to examine the industry-generated standards and licensing requirements described in Section 
IV. 1, since these specify from the employer's perspective how much education is enough for various positions. 
Another is to examine the debates within occupational fields about the intensity of postsecondary education, 
though this discussion can take place only in occupations where there is some kind of organization-either an 
industry association, a licensing body, or an association of instructors-to convene such a discussion. For the 
moment, the former seems an easier starting point. 

I 1. 3: Support Services 

Effective programs provide a variety of support services, as appropriate given the needs of their students. 

While PSOE may be principally an educational program, an institution that fails to consider the need for 
support services may have low completion rates and fail for that reason alone. In community colleges and other 
forms of PSOE, five kinds of support services seem particularly important for students in occupational 
education: 

• Guidance and counseling mechanisms: Particularly for "experimenters" unsure of what they want to 
do, or dislocated workers looking for alternative occupation, or welfare recipients seeking a way back 
into the labor force, guidance and counseling is particularly important. Guidance and counseling 
mechanisms could include introductory courses or project-based efforts to get students to learn about 
occupational alternatives and the efforts of occupational instructors themselves, and not simply the 
efforts of guidance counselors in their offices. The Puente program in California, which creates learning 
communities of instructors along with counselors, provides another interesting model of integrating 
counseling into instructional programs. 

However, as in high schools, there are many complaints about the adequacy and type of counseling available in 
community colleges and other postsecondary institutions; counselors are often criticized for providing 
information solely about academic issues and transfer, neglecting career-oriented counseling. While instructors 
and administrators can describe what happens within their own institution, there seems to be no central source 
of information about postsecondary guidance and counseling, no established conception of the developmental 
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processes involved in adult decisions about careers, no conventions about good practice. NAVE could provide 
a substantial service to PSOE by carrying out, perhaps as part of case studies (IV.4), an analysis of what's now 
available in postsecondary guidance and counseling and some method of developing conceptions of good 
practice. 

• Remedial/developmental education: The problem of underprepared students in all postsecondary 
programs has grown substantially, and PSOE is no exception. Community colleges and technical 
institutes typically provide some forms of remedial or developmental education, though the type and 
quality of these programs varies enormously, and almost nothing is known about the effectiveness of 
these programs in helping students complete their goals. In the best cases, individual instructors have 
developed their own approaches (often constructivist) through trial and error, or a developmental studies 
department has devised a coherent policy, or instructors have linked developmental courses with 
occupational courses in learning communities. (See also 1.2 above, on integrating academic and 
occupational education for remedial purposes). In others forms of PSOE, remedial education may be 
missing, or students may be directed to adult education programs of varying (and mostly abysmal) 

quality.^ 

The nature and effectiveness of remedial/developmental education is a large subject, applicable to most 
postsecondary institutions, and this is not an issue that NAVE can readily research on its own. As with many 
other questions, the case studies of local institutions (see IV.4) provide one opportunity to examine the nature 
of these efforts, their relationships (if any) with occupational programs, and (if institutions have good enough 
data) their effectiveness. However, NAVE might also explore the possibilities for joint research with other 
government agencies and centers concerned with the quality of postsecondary education. 

• Retention programs: The issue of who is a completer, and whether completion as defined by 
certificates and associate's degrees is important, is one of the most vexing questions in PSOE, one that I 
examine more fully in Section IV. 1. For the moment, however, it is sufficient to point out that 
community colleges have made various efforts to increase retention and completion. They can be 
categorized as sorting, supporting, connecting, and transforming strategies (Beatty-Gunther 1994), and 
the best community colleges use a variety of them. Again, institutional case studies (see IV.4) provide 
the easiest ways of learning more about the nature of these efforts and their interactions with 
occupational programs more specifically. 

• Placement efforts: In occupation-specific forms of education, the benefits may be lost unless an 
individual finds employment related to his or her field of study. Results from national data confirm that 
the economic returns to PSOE are much higher when individuals find related employment, especially 
among women (Grubb 1997 or 1999); and some preliminary California results based on UI data 
corroborate these findings. Therefore, occupational education by itself may be necessary but not 
sufficient to gain access to well-paid employment, and placement efforts may be necessary as well. 

Again, local case studies provide low-cost ways of examining what community colleges and other local 
PSOE programs now offer. 

• Services for students with disabilities: Because community colleges are open-access institutions with 
relatively low spending levels, the diagnostic procedures to identify students with learning disabilities are 
often weak, and services to help such students are often inadequate. Community college instructors 
report that they suspect having students with learning disabilities, but do not know how to help them and 
have nowhere to refer them. Such students often end up in remedial classes, which may not be 
appropriate. While I am not the right person to recommend how to address this issue, a fuller analysis of 
support services would consider carefully the availability of such disabilities and their effects on the 
progress of the students themselves as well as their effects on other students. 



O 




13 



II.4: Connections to Other Programs 



Effective programs provide their clients or students with pathways or "ladders" of further education 
opportunities, so they can continue their education and training when they are able to. One issue of creating 
such pathways in PSOE involves the link between high school and community college, in the tech-prep 
programs described in Section 1.3 above. The other common linkage is that between community colleges and 
four-year colleges, in articulation mechanisms to smooth the process of transfer. In many occupational 
areas-e.g., business, computers and information technology, engineering technology, some health 
occupations-the possibilities of transfer are substantial, and both instructors and institutions provide 
encouragement and support (like transfer centers); in other areas — automotive areas, metal trades, 
construction trades, cosmetology — there are no comparable baccalaureate-level programs and so articulation 
mechanisms are scarce. The possibility of and support for transfer should be one of the issues incorporated into 
local case studies (see IV.4). 

However, the most important kinds of connections for NAVE to investigate are probably the connections 
among different types of occupational education and training programs. These connections — the efforts to 
create more coherent systems of work force development encompassing short-term job training, adult 
education, welfare-to-work programs, training for incumbent workers, and occupational education in 
postsecondary institutions — have been the subject of substantial research by NCRVE (see Grubb et al. 1999, 
for the latest effort) as well as research by the earlier NAVE (e.g., Hollinger and Harvey 1994). However, 
changes in federal legislation — particularly the Workforce Investment Act (WIA) of 1998 and the changes in 
welfare legislation passed in 1996 — have substantially changed the conditions within which local programs 
operate, and therefore new research may be warranted. In particular, the emphasis of both welfare legislation 
and WIA on "work first" conceptions-in which individuals are first placed in employment without worrying 
about their competencies — contradicts the human capital emphasis of PSOE, and the resolution of this 
conflict is a central issue. 

The issues involved in relations with other programs include the provision of services and special programs for 
welfare recipients (if any); the ways PSOE institutions participate in state and local Workforce Investment 
Boards; the incorporation of PSOE into information and services provided by local one-stop centers, which 
play a central role under WIA; the eligibility of PSOE for the Individual Training Accounts (ITAs) provided 
some individuals, the numbers and types of individuals who actually use ITAs to attend community colleges 
and other forms of PSOE; and the difficulty PSOE institutions have in meeting WIA accountability measures 
(see 1.1 above). The period within which NAVE will operate-with data being collected roughly between 
summer of 1999 and summer of 2001 — will be one of substantial turmoil, as states create Workforce 
Investment Boards under WIA and begin planning local efforts, and therefore it may be possible only to 

understand how PSOE is participating in this process and not what the outcomes are. ^ 

One way to carry out this research is again to incorporate questions about participation in WIA into the 
questionnaire surveys (see IV. 3) and local case studies (IV.4). However, because these issues involve 
interactions between PSOE and other programs, it would be far better to conduct research examining WIA 
efforts and welfare programs as well as PSOE. This can be accomplished with community case studies, where 
several communities are identified and all the programs within these communities providing PSOE, job 
training under WIA and welfare, adult education, and other forms of work force development are examined to 

learn about their interactions (see IV.5).-^ In this way the perspectives of each program about the others can be 
ascertained, rather than viewing the cooperation among programs solely from the vantage of community 
colleges and other PSOE institutions. Of course, the communities selected for such studies could be a subset of 
the communities in which the institutional case studies (see IV.3) are conducted, so that the community cases 
studies are effectively extensions of the institutional case studies. 
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II. 5: Data and Improvement 

Effective programs collect appropriate information about their results and use these to improve their Quality. In 
the case of PSOE, these informational issues include the development of accountability measures, examined 
above in Section 1.1, and the development of state and local data on employment outcomes, examined in the 
next section. 

II. 6: Outcome Measures: National and State Sources 

To many researchers and policymakers in particular, outcome measures-rather than the elements of program 
structure I have covered in Sections II. 1 to II.5-are the only reliable indicators of effectiveness. The good news 
is that better data have become available in the last ten years, both from national data sets and from state data 
sets based on UI wage record data. 

In terms of national studies, I have recently surveyed the available literature for the Community College 
Research Center (Grubb 1999), and I see no reason for replicating that survey. There are, however, some topics 
that have not been extensively studied, and if NAVE wants to invest in further analyses of national data, it 
should concentrate on these issues. One of them is the problem (see also IV. 1) of whether students who 
complete small amounts of PSOE benefit or not; this requires disentangling the effects of field of study, the 
relatedness of education and subsequent employment, and student intentions (i.e., for upgrade training) on 
economic returns, since it's likely that all of these influence the value of small amounts of education. A second 
issue that merits more exploration is the effect of finding education-related employment, since this has been 
investigated in only one study (Grubb 1997); there are obvious selection and self-selection effects to consider, 
as well as alternative ways to define education and employment as being related or not. Third, the effect of 
local economic conditions and of the business cycle on the employment benefits of sub-baccalaureate education 
have not been studied, though they are arguably quite important. A study reanalyzing certain data sets — 
particularly the Survey of Income and Program Participation and the National Longitudinal Study of Youth — 
could address these unanswered questions (see IV.8). 

States have made great progress in developing data based on UI data. A few states have been leaders in these 
efforts, including California, Washington, Texas, Florida, and North Carolina. However, the analyses they have 
carried out so far are relatively unsophisticated (partly for political reasons), and have generally been confined 
to a few cross-tabulations. One valuable NAVE project would therefore be to take one or more of these state 
data sets and carry out more sophisticated multivariate analyses (see IV. 10). In addition to carrying out such 
analyses, such a study should consider how to translate complex findings into simpler terms, since 
policymakers and administrators are unable to read regression coefficients. In addition, I propose in Section 
IV. 1 using these state data, in conjunction with other information, to carry out further analysis of the complex 
issue of completion. 

One other use of state UI data would be to conduct an outlier study. In the school effectiveness literature, 
particularly effective (and ineffective) schools were identified with regression analysis of test scores controlling 
for family background and school resources; schools with high positive residuals were then considered 
especially effective, and case studies were conducted to see how they differed from other schools. Similarly, in 
the realm of job training, Chris King and his colleagues (King, Lawson, Olson, Trott, and Baj 1998) selected 
especially effective JTPA programs in Texas and Illinois based on earnings over 155 percent of poverty and on 
continuous employment. The outliers they visited were committed to a structured process for determining high 
demand and emerging occupations; they emphasized occupational skills training rather than short-term job 
search assistance and on-the-job training; they stressed certain support services, including intake, assessment, 
counseling, and case management continuing throughout the program and afterwards; and they constrained the 
choices of training activities, referring individuals only to approved providers and only in high-demand or 
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